out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB by ShelbyZ · Pull Request #11848 · fluent/fluent-bit

ShelbyZ · 2026-05-26T17:33:39Z

Increase size from 5MB -> 10MB to match recent changes with PutRecords API

Kinesis Streams PutRecords API change in 2025 to support events up to 10MB (ref) while our defaults reflect the older value 1MB.

Add runtime tests to validate the new limit:

event_size_near_limit: Validates events just under 10MB are accepted
event_size_at_aws_max: Validates events near AWS max are discarded (existing behavior)
event_truncation_with_backslash: Validates backslash handling at truncation boundary

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

[N/A] Example configuration file for the change
[N/A] Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Logs: https://gist.github.com/ShelbyZ/0c8d005d39ddc8759cf6b77d7dc90dc5

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[N/A] Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- Increased Kinesis output plugin maximum payload size from 5 MB to 10 MB.

- Increase size from 5MB -> 10MB to match recent changes with PutRecords API Signed-off-by: Shelby Hagman <shelbyzh@amazon.com>

coderabbitai · 2026-05-26T17:34:01Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5d606f0-5737-407e-8e84-505ff3f5a545

📥 Commits

Reviewing files that changed from the base of the PR and between c2fec52 and cf8e1b4.

📒 Files selected for processing (1)

tests/runtime/out_kinesis.c

📝 Walkthrough

Walkthrough

This PR increases the Kinesis plugin's maximum PutRecords payload size from 5 MiB to 10 MiB and adds three runtime tests that validate event-size boundary behavior: an event just under the limit, an oversized event expected to be discarded, and an oversized event with backslash escapes to exercise special-character handling.

Changes

Kinesis Payload Size Increase and Event-Size Boundary Tests

Layer / File(s)	Summary
Kinesis PutRecords payload size constant update `plugins/out_kinesis_streams/kinesis_api.h`	The compile-time constant `PUT_RECORDS_PAYLOAD_SIZE` is doubled from 5,242,880 (5 MiB) to 10,485,760 (10 MiB), raising the maximum payload for PutRecords API operations.
Kinesis event-size test infrastructure `tests/runtime/out_kinesis.c`	Add Kinesis API header include to expose the payload size constant. Define `create_large_log_event` helper to allocate and build JSON records of a requested size, and `run_kinesis_test_with_data` runner that initializes a Kinesis output, pushes test data, and tears down the engine context.
Kinesis event-size boundary test cases `tests/runtime/out_kinesis.c`	Add three runtime tests covering payload boundaries: `flb_test_kinesis_event_size_at_limit` for an event just under the maximum, `flb_test_kinesis_event_size_over_limit` for an oversized event, and `flb_test_kinesis_event_size_with_backslash` for an oversized event with regular and boundary-adjacent backslash escape sequences. Register all three in TEST_LIST.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related issues

Enhancement: Make kinesis_streams MAX_EVENT_SIZE configurable or support >1MB records #11524: Addresses the Kinesis plugin's hard-coded maximum record/payload size constraint by raising PUT_RECORDS_PAYLOAD_SIZE to 10 MiB, directly related to the request for support for >1 MB or configurable maximum sizes.

Suggested reviewers

edsiper
cosmo0920

Poem

🐰 From five to ten the limits grow,
Kinesis streams now free to flow,
With tests that dance on boundaries near,
Backslashes skip without a fear,
A doubling done, the code runs clear! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: increasing PUT_RECORDS_PAYLOAD_SIZE from 5MB to 10MB in the out_kinesis_streams plugin.
Docstring Coverage	✅ Passed	Docstring coverage is 84.62% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/runtime/out_firehose.c`:
- Around line 789-793: The variable declaration "size_t boundary" must be moved
from mid-block into the top of the function’s declaration section; declare
"size_t boundary = MAX_EVENT_SIZE - 1;" alongside the other local variable
declarations at the start of the function, then use that variable in the
existing logic that sets large_json[boundary] and large_json[boundary + 1] when
checking "if (boundary + 1 < total_len - suffix_len)". Ensure no other
references rely on a narrower scope so compilation and behavior remain
unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a2b7b2f1-1a61-4a5a-88ec-61e525b12e75

📥 Commits

Reviewing files that changed from the base of the PR and between f6126eb and 673f5b2.

📒 Files selected for processing (2)

plugins/out_kinesis_streams/kinesis_api.h
tests/runtime/out_firehose.c

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 673f5b201e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Signed-off-by: Shelby Hagman <shelbyzh@amazon.com>

out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB

690fb72

- Increase size from 5MB -> 10MB to match recent changes with PutRecords API Signed-off-by: Shelby Hagman <shelbyzh@amazon.com>

ShelbyZ requested a review from a team as a code owner May 26, 2026 17:33

github-actions Bot added the docs-required label May 26, 2026

ShelbyZ temporarily deployed to pr May 26, 2026 17:33 — with GitHub Actions Inactive

ShelbyZ had a problem deploying to pr May 26, 2026 17:33 — with GitHub Actions Failure

coderabbitai Bot reviewed May 26, 2026

View reviewed changes

Comment thread tests/runtime/out_firehose.c Outdated

chatgpt-codex-connector Bot reviewed May 26, 2026

View reviewed changes

Comment thread tests/runtime/out_firehose.c Outdated

Comment thread tests/runtime/out_firehose.c Outdated

ShelbyZ changed the title ~~Kinesis streams increase~~ out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB May 26, 2026

ShelbyZ force-pushed the kinesis-streams-increase branch from 673f5b2 to c2fec52 Compare May 26, 2026 19:11

ShelbyZ temporarily deployed to pr May 26, 2026 19:11 — with GitHub Actions Inactive

ShelbyZ temporarily deployed to pr May 26, 2026 19:31 — with GitHub Actions Inactive

tests: out_kinesis: Add event size limit tests

cf8e1b4

Signed-off-by: Shelby Hagman <shelbyzh@amazon.com>

ShelbyZ force-pushed the kinesis-streams-increase branch from c2fec52 to cf8e1b4 Compare May 26, 2026 20:33

ShelbyZ temporarily deployed to pr May 26, 2026 20:33 — with GitHub Actions Inactive

ShelbyZ temporarily deployed to pr May 26, 2026 20:53 — with GitHub Actions Inactive

edsiper merged commit ff1ed11 into fluent:master Jun 4, 2026
50 of 53 checks passed

edsiper added this to the Fluent Bit v5.0.7 milestone Jun 4, 2026

BrewTestBot mentioned this pull request Jun 6, 2026

fluent-bit 5.0.7 Homebrew/homebrew-core#286580

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB#11848

out_kinesis_streams: increase PUT_RECORDS_PAYLOAD_SIZE to 10MB#11848
edsiper merged 2 commits into
fluent:masterfrom
ShelbyZ:kinesis-streams-increase

ShelbyZ commented May 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 26, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Suggested reviewers

Poem

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ShelbyZ commented May 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Suggested reviewers

Poem

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ShelbyZ commented May 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 26, 2026 •

edited

Loading